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Introduction 


This is a report of a consultation event run by the NCCPE on September 14" 2018. Over 80 
delegates joined us to consider the current consultation on the draft REF guidance. The event 
closed with a panel discussion between: 

e Steven Hill (Head of Research at Research England chair of the REF Steering Group) 

e Imran Khan (Head of Public Engagement at Wellcome and member of REF Main Panel A) 

e Sophie Duncan, NCCPE (chair) 


The NCCPE circulated a briefing paper in advance which can be accessed here. 


A series of table discussions addressed the following topics: 

e Equality and diversity: the briefing paper identified the absence of references to Equality 
and Diversity in the draft documents. Tables were invited to discuss how best to address 
this 

e Rigour and robustness of evidence: the briefing paper outlined our proposal to highlight 
the importance of rigour in assessing impact. Tables considered this and fed back on the 
suggested framing and wording of the response 

e Consistency and coherence: four tables reviewed the NCCPE’s suggestions for how more 
consistency and clarity could be achieved particularly in relation to Annex A of the Draft 
criteria, which lays out a set of exemplar indicators of impact 

e Capacity building: tables stood back from the details of the consultation to consider the 
capacity building challenges linked to the REF, and to feedback on the NCCPE’s suggestions 
of how they might offer tools and training in this area 


Executive Summary 
Delegates provided invaluable feedback on the NCCPE’s proposed response. 


e There was broad agreement that the guidance about Equality and Diversity needed to be 
extended beyond internal staff policies to include the approach to impact. There should be 
a clear expectation set by the panels that equality and diversity will be addressed in impact 
planning (in the ‘environment’), and within impact case studies. Some risks with this were 
pointed out, for instance that it might encourage superficial or tokenistic behaviours. 

e There was strong support for asking for stronger guidance on what would count as ‘robust 
evidence’ and on the important role of evaluation. But there was a very mixed response to 
the suggestion that assessment of the ‘rigour’ of the process of engagement might be 
foregrounded in the guidance, given that the REF is focused on assessing the outcomes and 
impacts arising from research (not the engagement process itself). However, many 
delegates wanted to see more clarity being offered about how ‘pathways’ to impact will 
be judged by panels as part of the assessment process. 

e The NCCPE’s suggestions about how the guidance about indicators of impact might be 
made more consistent and coherent were welcomed by many, with a number of very 
useful and practical suggestions about how to develop the draft table of indicators 


proposed in the NCCPE’s discussion paper. There was broad agreement that categorising 
indicators as ‘structure’, ‘process’ or ‘outcome’ indicators is helpful. 

e Discussion of capacity building highlighted how much support will be needed, both by the 
panels, and by the sector, to ensure a more rigorous approach is embedded for REF 2021. 
The key ‘pressure points’ for addressing capacity building effectively are summarised in the 
table below: 


Building capacity for public engagement (PE) in REF 2021 


The particular barriers / e Issues of resources (time, funding, skills etc.) 

blocks to the integration e Structural issues (e.g. lack of strategic investment in 
areas like reward and recognition or long term 
investment in capacity building) 

e Issues of language/definitions 

e The challenge of evidencing impacts from PE 

e (Perception) PE not valued/seen as risky 

e The need for additional guidance/examples 


of high quality public 
engagement in REF 
submissions 


The ‘triggers’ or methods |e Reward, recognition, career progression 
you have deployed which |° Training/researcher development 

e Champions/leadership 

e Communication/framing 

e Funding 

e Support 

e Case studies and examples 


have been successful 


Ideas for how the NCCPE e Support to panels 

and other agencies might e Support to sector/institutions 

e Support to PEPs 

e Training/CPD 

e Advocacy 

e Setting the standards/providing examples 


provide well targeted 
support 


Next steps 


The NCCPE has revised its consultation response in the light of the feedback received at the event 
and has published a version of the response which we will submit on 15* October. 


We will be using the feedback on capacity building to inform our plans for our ongoing support 
activity. 


We would like to thank all the delegates for their input, and the various people who contributed 
feedback on the discussion paper which underpinned this event. 


Introduction and Overview 


The NCCPE provided a short overview of their approach to the current consultation. The 


presentation covered the following points: 


Engaging publics with research th Š l ul 


Discussion Paper: Assessing impacts 
arising from public engagement with re 
After the REF - Taking Stock: summary of feedback 
research 
= Reviewing the REF 2014 impact case studies and templates 
2012 2014 2016 


Building on Success and 
Leaming from Experience 


REF REF 


February July - October 
2018 2018 


A summary of the various 
reports and briefing papers 
prepared by the NCCPE over 
the last 6 years, all of which 
can be accessed here 


Headlines 


* Public engagement is pervasive: 3108 of the 
6640 case studies (47%) made some reference 
to engaging with the public. 

* Public engagement in the REF is nearly always 
focussed on changes to understanding and 
awareness. Much more rarely is it 
foregrounded as a route to realising legal, 
technological or commercial impacts or more 
instrumental outcomes. 


Search REF Impact Case 


; i J . Studies 
* Evidence provided of impact on public Browse the index below or search all Case Studies using keywords 


understanding and awareness is often weak. 


* Case studies featuring PE scored similarly to 
those that didn’t 


The key findings from the 
NCCPE’s review of the 2014 
Impact Case Study database 


Your reactions to the issue we have 
Equality and diversity highlighted and our proposed 
approach 

Your suggestions for how to focus 


Rigour and robustness the response — including resources, 


of evidence tools, frameworks, other evidence 
we should be citing 
Consistency and PERNE ee 
h Questions, risks, uncertainties which 
coherence we should take account of as we 


finesse our response 


The headline issues which 
the NCCPE plans to 
foreground in their response 
to the current consultation, 
and which were outlined in 
the briefing paper circulated 
before the event 


Table discussion responses 


Following this presentation, tables were invited to record their responses to a set of prompt 
questions relating to the three discussion topics for discussion. The notes recorded by each table 
have been transcribed and clustered in the following tables. 


Equality and diversity 


The briefing paper identified the absence of references to Equality and Diversity in the draft 
documents. Tables were invited to discuss how best to address this 


Whilst there was broad agreement that this issue should be foregrounded in the NCCPE’s 
response, there was some caution about setting unrealistically high expectations of how the sector 
might respond. 


Do you agree that this is a significant issue which we should prioritise in our response? 


Strong support for addressing the issue 
e Increasing the reference to equality and diversity, particularly audiences and 


communities, can only help with the quality of public engagement work, REF-related or 
otherwise 


e Yes 

e This hard to do, so reward robust methods in doing this even if the level of impact is a 
little less than others 

e 100% of our table agree yes 

e This is an opportunity, if prioritised, to re-orient research to those who would benefit 
most because they are underserved 


But risks of encouraging inappropriate and unprofessional behaviours 

e The danger is that the community group should be appropriate, you can’t force it 

e Concerns about parachuting into communities who we’ve not seen before and then 
leave again 

e Including ‘community voice’ is difficult and sensitive; can involve vulnerable people 

e Ethics and equality: make sure that focus on rigour doesn’t lead to badgering vulnerable 
impact beneficiaries (e.g. mental health research) for evidence. This would make their 
situation worse 

e Diversity and equality as part of the training of researchers on the moral choices and 
consequences of research and its impact on society 


And of other unintended consequences 
e We are concerned that this could lead to people thinking that if they don’t engage with 
disadvantaged communities it’s no good: impact is not as good 
e The research community may shout ‘what you’re telling me this now?! I’ve only just got 
my head around public engagement’ 


e There isn’t consistency across all impacts: what does this mean for policy and business et 
cetera 


What key points should we emphasise (in terms of case study submissions and environment 
statements)? 


Encouraging panels to be explicit that they will value activity that contributes to E&D 
e Equality and diversity should be a factor throughout (i.e. Partnerships and 
collaborations) as well as processes and outputs 
e Anacknowledgement that engagement with more diverse communities and audiences 
can be more complex and demanding and should be recognised as such 
Lots of useful suggestions for additional practical guidance 
e Can we define and unpack ‘reach and significance’ and give examples? 
e How do we capture the diversity of stakeholders? 
e Consider how it relates to reach. 
e Breakup the general public into a mosaic of suggested groups to encourage thinking of 
diversity, and feed this into ‘reach’ 
e How it relates to process: the where, the who, how and non-native English speakers 
e Start with stakeholder and audience mapping to identify appropriate publics. Ensure 
there is diversity and equality within 
e Most HEl’s have equality and diversity at the vision level, it’s the same for the REF 
e Importance of recognising the process as much as the outcomes of research 
e Reach an equal and diverse audience by producing a range of targeted outputs, for 
instance to reach those who don’t read journals 


Are there any frameworks / guidelines policies / examples, which you think we should 
reference? 


e Involve’s patient guides 
e Educational organisations’ guidance and policies 


e BBC audience categories 

e Museum audience classifications 

e Emily Dawson at UCL would be a good person to consult 

e National standards for community engagement see voicescotland.org.uk 

e Scottish government national outcomes frameworks 

e One table suggested the following amendment to section 4 of the Environment 
template: ‘This section should provide information about the submitted unit’s research 
collaborations, networks and partnerships, their equality and diversity, including 
relationships with key research users, beneficiaries or audiences’ 


Rigour and robustness of evidence 


The briefing paper highlighted the importance of addressing rigour in assessing impact. Tables 
considered this and fed back on the suggested framing and wording of the response. 


There was considerable ‘push back’ on the suggestion that ‘rigour’ might be added to ‘reach and 
significance’ as a key criterion for assessing impact. However, there was support for providing 
clearer expectations about the provision of robust evidence arising from evaluation, with a range 
of helpful suggestions for how this could be framed. 


Do you support the case for adding ‘rigour’ to ‘reach and significance’? Why / why not? Could 
the wording be improved? 


A very mixed response 


Half of the table said yes and half of the table said no 

Disagreement: this approach doesn’t account for accidental engagement, we should 
assess the change made not the process to make that change, this is the wrong platform 
for assessing rigour: REF is measuring change rather than the process to get that change 
Concerned with introducing the element of rigour at this stage in the REF process 
Introducing rigour would also cause concern for those case studies with strong impact 
but that struggle to evidence rigour as this may have been approached by other actors in 
the research process 

Rigour of pathways to impact can’t be assessed with the same weighting as reach and 
significance of impacts themselves 

How can the rigour of accidental impact be measured? 

This would be a significant burden and it’s too late to add another element the 2021. 
Perhaps the 2028 

Only rigorously carried out engagement activities are likely to have the biggest impact. 
However there can be very rigorously carried out public engagement programs that do 
not achieve impact, and these should not score as highly 

Rigour: more robustness of evidence rather than more rigour might generate a whole 
industry behind evaluation 


Beware of unintended consequences 


It is vital to bear in mind that these guidelines will ultimately influence the types of 
public engagement activities carried out, so think what activities do we want to 
encourage and what guidance will lead there 

There are concerns about providing obstacles and deterrents to staff who might consider 
submitting a case study 

Emphasising rigorous evaluation requirements within REF guidance could create a 
hierarchy of evidence which may counter against the key REF goal to identify impact 
from excellent research however it may arise 

It is important to show that all levels of impact are important and valid, to dissuade 
researchers from making heroic and grand assumptions about impact because that’s 
what they think is wanted or awarded 

Concerns about how this will be interpreted by researchers, and what type of 
engagement activities it will end up encouraging 


Other considerations 


You don’t need to see the detail of the evaluation process 

Panels should ensure that this is considered and it should be fundamental to what’s 
going on anyway and what should be expected 

A radio four interview with high reach but poor quality evaluation should not score 
highly 


Alternative approaches to the challenge 


Suggestion: instead of rigour and evaluation put the emphasis on producing and 
demonstrating high-quality robust evidence. Make this consistent across panels and 


provide a level playing field while allowing for different subjects producing different 
kinds of evidence 

Include prompts and guidance to include an embedded evaluation e.g. As an example of 
robust evidence: this could help with better understanding how research leads to impact 
Rigour is something that should come across in the environment element of the REF and 
is not necessarily something that should be assessed individual case study level 

Rigour is a loaded term which may mean different things to different research areas. But 
REF leads the agenda on quality improvement so it’s important that rigour is recognised. 
The sentiment of rigour without using that term would maybe be a good first step 

| support it but can see it might be difficult certain units of assessment. Perhaps rigour in 
some of the range of areas i.e. in the process of setting up evaluating or responding 

| think the paragraph 293 of the consultation of draft panel criteria goes far enough in 
introducing rigour and robustness to case studies. It mentions integrity and the expert 
judging of this. It also follows that strong methods and rigorous evaluation lead to highly 
rated indicators and evidence 

Can we have assurances from the panels that they will judge evidence in a robust 
manner (which perhaps didn’t happen in all cases in 2014) 

Some people who said no, did feel that it should be emphasised more but not 
foregrounded alongside reach and significance 

Concerns about how rigour would be interpreted. It needs to be flexible 


Do you think we should foreground the role of evaluation? Are there any risks with doing 
this? What points should we emphasise in making the case? 


A variety of concerns were expressed 


Tension between internal and external evaluation and overplaying the importance of this 
Evaluation needs to be appropriate and credible, so is that basically ‘robust’? 

There are risks in foregrounding evaluation: it might discourage initiatives that are 
difficult to evaluate rigorously in the short term 

How do the panels know what good and meaningful impact looks like? How can the 
rigour be articulated to them? 

Does including rigour for public engagement alter the benchmark for other types of 
impact? 

There isn’t a big enough discussion being had about what robust evidence is in order to 
produce such guidance 


Other helpful clarifications 


Steven Hill’s point about unplanned impact: the guidance should suggest and encourage 
exploring it after the event, rather than not at all 

Assessment of process can be discussed and looked at under the environment rather 
than impact 

Serendipitous impact won’t necessarily be able to claim this, but planned public 
engagement should have aims that can be tested 

Evidence of reach and significance needs to improve and REF leads the agenda, so yes, 
more and evaluation with caution 

Yes, evaluation is key to knowing what impacts have been achieved, and good evaluation 
should also capture serendipitous outcomes, events and impacts 


e Building rigour and good evaluation is perhaps more timely at the start and throughout a 
REF cycle and many projects for REF 2021 will be complete too late to change strategy or 
methodology 

e Yes, this is important to emphasise rigour. Need also to highlight that impact may be 
intangible but should nonetheless be robustly evaluated 

e ls it always important to highlight the evaluation process in evidencing impact? Is the 
strength of the impact demonstrated in the case study not linked to the evaluation 
process anyway? It’s important to recognise the role of evaluation in the process but not 
necessarily highlight it as an explicit output 

e The rigour of the research is ensured by the two star threshold of criteria. However the 
rigour of the method behind constructing the impact claim cannot be articulated within 
the case study format. By adding a method section, i.e. What methods were used to 
create the impact claim in a hundred words would allow for rigour of the claim 

e Some of the group feel that this is been addressed already in the guidance through the 
routine submission of strong evidence. They should be checks in place with this in terms 
of the organisational scrutiny and overview and the assessment by the panels 

e Others felt that it should be more foregrounded, possibly even to the point of including a 
methodology section to the case study how the impact was evidenced 


Are there any frameworks / tools / resources which you think we should reference? 


e Provide robust examples of evidence that could be used consistently across panels 

e We would like to see more and clearer guidance on what makes a four-star case study 

e Guidance on what robust evidence looks like: concrete and specific examples, for 
instance case studies 

e Potential source of tools would be the work of academics in the field of robust evidence, 
for instance Emily Dawson 

e Mixed methods research methods, going beyond the instrumentalised social health 
frameworks 

e Scottish government has some good guidance on standards of community engagement 
which covers 10 areas including planning, evaluation, processes, people, inclusion. | was 
really impressed and it can be applied elsewhere 

e NCCPE Evaluation resources and Guides 

e R Brauer (2018) What research impact? Tourism studies and the changing UK research 
ecosystem 

e Fast track impact has a range of publications resources and tools 


e The Emerald impact literacy workbook 


Consistency and coherence 


Four tables reviewed the NCCPE’s suggestions for how more consistency and clarity could be 
achieved, particularly in relation to Annex A of the Draft Criteria, which lays out a set of exemplar 
indicators of impact 


There was strong support from many people for the suggested use of ‘structure, process and 
outcome indicators’ as a way of providing a more coherent definition of indicators of impact. 
There was also considerable support for the NCCPE’s suggested re-framing of the draft Annex A’s 


areas and indictors of impact. Anumber of people suggested that the NCCPE’s proposed area 
‘Education, Lifelong Learning and Community Development’ be split into two. 


While there was significant support for ‘distributing’ public engagement across the guidance 
(rather than containing it in one discrete area) risks with this approach were identified. 


Areas of Impact 
e Do you agree with our suggested re-framing? Would you add anything? 
e Are there risks / unintended consequences with doing this? 


Reflections on the proposed new area. ‘Impacts on education, lifelong learning and 
community engagement’ 

e Impacts on education, lifelong learning and community engagement: does this reflect a 
sector? 

e Adda10" area of impact: impacts on citizenship. The motivation being to improve 
participation in civic society, informal engagement in decision-making and considered 
involvement in public/online discussion 

e Consider separating education and lifelong learning (skills related) from community 
engagement and citizenship (agency -related). See above. 

e Feedback the citizenship role into the environment template 

e Under skills — make a link to the industrial strategy 

e Alternative: impacts on understanding, skills and participation (worried we lose category 
or a catchall) 

e Community engagement or community development? 

e The new wording would help introduce more case studies from the humanities 

e = Skills or lifelong learning? 

e Impacts on understanding, learning and participation: is this underlying all the other 
areas of impact? 


Other reflections 

e The new wording is more in line with previous. More specific about outcome (noun) over 
process (verb) 

e It is good to have discussions but not to get too caught up in putting things in boxes, so 
this is valuable as a toolkit 

e The table has different responses: an example of how important consistency is, as all the 
table has different interpretations of the meaning of the word 

e Does this query have its roots in the history of development of public engagement? 
What is the difference between public engagement and community engagement? 

e Is there an issue of NCC politics to ensure public engagement is included in everything? 

e The definition of impact should be consistent 


Indicators 
e Do you find this approach helpful? Are there better alternatives we could use? 
e Are there risks / unintended consequences with doing this? 


Positive endorsement of the approach 
e The indicators are really good! Especially outcomes 
e It is helpful to have examples split into indicators 


e It is useful to have a document about the range of what impact can be 

e It helps to think about the process of research and how that might have an impact that 
you’ve not thought through 

e It helps think about the impact journey 


Enhancing and developing the table 

e With the draft table, you could helpfully add ‘mechanism’ or ‘approaches’ column 
between ‘public roles’ and ‘contribution’, to explore how the impacts might be achieved 

e How do these map onto the areas of impact? 

e How do these indicators align with the funders’ expectations and ideas around impact 
(for instance Wellcome versus pathways to impact etc.)? 

e It is a helpful grid to identify indicators, though the outcome indicators felt vague 

e There needs to be some rewording of the terminology. It needs more basic explanation 
as academics won’t understand some of the terminology 

e |t doesn’t seem to capture the measurement of academics changing practice 

e We're not sure how the table works: is it a flowchart or not? A few people found it 
difficult to follow 

e The table broadly felt that public engagement should be embedded throughout, that’s 
the aim 

e There wasn’t much about two-way and civic 

e Add in a clear definition of what structure, process and outcome categories mean 

e Add an executive summary and diagram of types of indicators 

e Provide narrative explanations 

e Inthe areas of impact, should animal welfare be with environment? 

e Separate community engagement into a separate area of impact or add into creativity 
and culture 


How might panels utilise this kind of content? 
e Who populates the panels that value these indicators? Perhaps more guidance and 
training to ensure all members have the same process 
e This table thought this approach was too complex and more suited for public 
engagement professionals over panellists 


Concerns about Panel A’s guidance on testimonials 
e Panel A says that they do not welcome testimonials offering individual opinions on 
evidence of impact. There are risks here — for instance with PPI and the role of patients 
on panels. If a patient wants to give personal testimony, where does this fit in? We need 
to ensure you don’t take the public voice out of the impact on the public. 


‘Dispersal’ of PE 
e Do you agree with our argument for dispersing PE? 
e Are there risks / unintended consequences with doing this? 
e What more should we do to flesh this approach out? 


Broadly speaking there was support for embedding public engagement across the framework, 
however a strong note of caution was raised that there may be unintended consequences, 
including a reduction in the number of case studies which focussed specifically on public 
engagement as a route to impact. 


‘Dispersed’ is not the best term, it’s too like evaporation. Perhaps ‘propagation’ or 
‘universalising’ 

Add in mechanisms or approaches - the how you do an activity or method - for each area 
of impact 

Potential risk: if public engagement is dispersed throughout then the risk may be that 
there are no stand-alone public engagement impact case studies therefore meaning 
public engagement could only be included as a bolt on 

Collectively we thought the idea of having public engagement throughout was a good 
thing. It would be helpful for individual researchers, the panels to assess and impact 
managers but we would caution that another word should be used instead of dispersal 
for instance: distribution, embedding, propagation, spread, universalising, integration, 
permeating 

Public engagement is the methodology and route to impact 

Could you lose the value of public engagement in itself as a result? 

You still need to evidence change 


Plenary feedback 


A short plenary session saw the following points highlighted by delegates: 


Equality and Diversity 


Concerns about highlighting E&D included the lack of best practice in the sector, and the 
risk of tokenism 

The need to start with identifying appropriate audiences 

The important role of the Environment statement in addressing this, for instance 
describing any investment in training to consider social and ethical consequences of 
research (and more broadly) 

It is important to take every opportunity to contribute to E&D, especially where taxpayers 
money is involved. 


rs 


Rigour 


It was noted that the guidance will affect types of PE that is done, and that this should be 
borne in mind. 

A concern about highlighting rigour and evaluation, and a strong preference for 
foregrounding the need to provide robust evidence instead. 

We’re in the business of education, so HEI’s should be rigorous in their internal processes. 
Can we say we’ll mark case studies down if the evidence is not robust? 

Strong concern about the suggestion of rewarding ‘method’ rather than ‘change’. 

Concern about the risk of failing to value unintended impacts. 

It was noted that panel members are not (usually) public engagement experts — so it will be 
important to consider how to make their jobs easier (for instance by offering training). 
Equally, we should not assume academics know what robust evidence of impact (and PE) 
looks like. 

Attribution is a critical issue: we need consistency in guidance about how to approach this. 
Consideration of process and approach is best captured in the Environment section. 


Consistency and coherence 


e The distinction between structure/process/outcome indicators was positively received 

e Comments on the table provided on page 9 of the briefing paper included that suggestion 
of adding an area of impact focused on citizenship 

e |t was suggested that a new column could be added to list examples of 
mechanisms/approaches of PE, to help to identify PE and express it. 

e The term ‘dispersal of PE’ was challenged — it sounds akin to evaporation. ‘Integration’ or 


‘embedding’ were preferred 


e The detail in the briefing paper was felt to be quite inaccessible. Can it be simpler? 


Capacity building to support REF 2021 


A short presentation by the NCCPE outlined the current support offered by the Centre linked to 


preparations for the REF. This includes: 


Institutional support 


A variety of training courses and CPD provision, including 


Introductory training courses (A Beginner’s Guide to PE 
and High Quality PE) 

Lessons from the REF: based on our review of the 2014 
case studies and including guidance on what is robust 
evidence 

Training on creating a fertile environment for 
engagement and impact 

Courses and masterclasses on evaluation 

Building Sustainable Partnerships training 
Consultancy and advice 


Individual support 


A Public Engagement Professionals Network to support 
PE professionals with access to CPD, peer support and the 


latest thinking on engagement 

The Engage Academy: a year-long professional 
development programme for PE professionals 

The Engage Researcher’s Academy: a year-long 
professional development programme for researchers 


with a passion for public engagement 


Longer term sectoral support 


Various briefing papers, reviews and analyses of how 
public engagement featured in REF 2014, listed here 
Support for institutional change, including our Engage 
Watermark benchmarking service 


The Engage Competition, to celebrate excellence in 
public engagement 

Research for All journal: a collaboration with UCL-IoE 
Press, a peer reviewed journal sharing cutting edge 
thinking about engagement and evaluation 


Delegates were invited to discuss three questions on their tables: 


e The particular barriers / blocks to the integration of high quality public engagement in REF 


submissions 


e The ‘triggers’ or methods you have deployed which have been successful 


e Ideas for how the NCCPE and other agencies might provide well targeted support 


Their responses are transcribed and clustered below. The key points arising in each area included: 


The particular barriers / blocks to 
the integration of high quality 
public engagement in REF 
submissions 


Issues of resources (time, funding, skills etc.) 
Structural issues (e.g. lack of strategic investment in 
areas like reward and recognition or long term 
investment in capacity building) 

Issues of language/definitions 

The challenge of evidencing impacts from PE 
(Perception) PE not valued/seen as risky 

The need for additional guidance/examples 


The ‘triggers’ or methods you 
have deployed which have been 


successful 


Reward, recognition, career progression 
Training/researcher development 
Champions/leadership 
Communication/framing 

Funding 

Support 

Case studies and examples 


Ideas for how the NCCPE and 
other agencies might provide well 
targeted support 


Support to panels 

Support to sector/institutions 

Support to PEPs 

Training/CPD 

Advocacy 

Setting the standards/providing examples 


Full notes of the table discussions are included as an appendix. 


Capacity building: plenary feedback 


A short plenary session saw the following points highlighted by delegates: 


Barriers 


e Ethics — getting ethical approval to do PE 

e Evidence — how to capture evidence of impact 

e We don’t know the rules until it’s too late! Guidance for 2028 is needed now — not 2021 

e Capacity within HEls to support the work we do (there are lots of temporary roles in this 
area, few full time contracts etc.). 


Enablers 


e It is helpful to have PE Professionals and REF/ Impact officers working closely together 

e There is a disconnect between Academics and Professional Service Staff, so having people 
in PE Professional or hybrid roles helps. 

e Citing economic value of impact case studies focuses people’s minds 


NCCPE 


e Work with Leadership Foundation and HEA (Advance HE) and with Vitae to ensure PE is on 
training programme for Senior Leaders in HEls 

e isthe word ‘Public’ a barrier? You could use ‘External’ instead 

e Provide really good examples of robust evidence of impact of PE 


Closing Plenary 


A closing plenary session saw delegates posing questions to our panel: 
e Steven Hill: Director of Research at Research England, and chair of the REF steering group 
e Imran Khan: Head of Public Engagement at Wellcome, and member of REF Main Panel A 
e Sophie Duncan: NCCPE (Chair) 


Public vs. external/policy engagement etc. — should we draw a 
distinction? 


The panel were asked to reflect on how helpful it is to focus on and identify ‘public engagement’ 
as a discrete area: 
e Wellcome ring fences PE funding to encourage engagement with a non-professional 
audience. The REF is not trying to be prescriptive, but to provide an easy way to describe 
(and value) all pathways to impact. 
e REF is about the impact not the pathway: but in practice different professional skill sets are 
required to do different types of engagement well, so it is helpful to differentiate. Too 
broad a definition can dilute professionalism. 


Definition of “Bodies of work”? 


The panel was asked whether a more precise definition of ‘bodies of work’ could be expected 
e It is very difficult to define — and we have learned that in cases like this you need to be 
either very specific or very open — a halfway house really doesn’t work. 


Panel A are a bit “sniffy” about testimonials and continued case studies 
— why is that? 


e There is concern that the panel might get lots of resubmitted ICS, which would be bad for 
the discipline area. 

e We expect that consultation responses will pick up on these differences between the 
Panels and feed them back. 


How much PE expertise is there in panels? 


e This is listed online in the details of panel members. Most panel members are not PE 
experts — so we need to make the guidance and advice accessible to non-experts. 


What about training for panel members? 


e = This will be guided by what the panels think they need. All will get unconscious bias 
training, and all with get specific guidance on certain things — e.g. citation data (where 
appropriate). 

e If a well-placed organization, for example the NCCPE, should produce guidance on quality 
public engagement, that is likely to be something the panels would consider. 


Research impact — PE — KE — Discuss. 


The panel were invited to reflect on the overlaps between knowledge exchange, impact and public 
engagement: 


There is lots of discussion about knowledge exchange/knowledge mobilization at the 
moment. PE could potentially be considered a process of KE, but also can extend the 
legitimacy and link of research to society. 

There are risks in subsuming PE activity within KE as HEI KE infrastructure tends to focus on 
business. 

It is important to emphasise that REF should not dominate conversations about public 


engagement. It is vital that HEls engage with society for all kinds of reasons — not just to 
realise research impact. 


REF/KEF — are they collaborating? If so how? 


The Knowledge Exchange Framework (KEF) is currently in development. What is the relationship 
between the KEF and the REF? 


There is a lot of cross-fertilisation. The response to the KEF consultation suggested that PE 
should be included in the metrics for KEF, but it’s hard to come up with metrics that don’t 
skew it and people argued strongly for a narrative element to KEF. This is being explored at 
the moment 

The NCCPE responded to the KEF metrics consultation. The response argued that the KEF is 


a place where all forms of PE might be reflected, not just those linked related to research 
impact. 


Can we discuss and decide the guidance for REF 2028 earlier? 


We need to evaluate REF 2021 before we press ahead with REF 2028. We talk about how 
important it is to evaluate PE. Evaluation of the REF process is as important. 


Research impact and PE environment 


Imran challenged delegates to think about how the Environment section could be used to 
emphasise and foreground the contribution of PE to research quality 


Is there a danger that we create a ‘publish or perish’ equivalent with 
impact? 


The REF celebrates diverse approaches to impact — PE, business, KTPs, policy, etc. It isn’t 
prescriptive — and the last REF did a good job of demonstrating this diversity. 

The REF tries for a balanced approach — it has reduced the emphasis on individuals, and on 
individual outputs. Impact case studies have always been about the unit not the individual. 


Anything else? 


The panel were invited for final reflections. 


Imran highlighted that: 

e Public Engagement is a pathway to impact, not impact. This is a good thing from 
Wellcome’s perspective. 

e The Environment section is really important — it’s only 15% of the overall score but it’s a 
really good opportunity to foreground the contribution of public engagement. 

e We're still in consultation and we really want to hear what you like and don’t like, backed 
up with examples. Make sure you also respond to the guidance on the Environment 
section. 


Steven highlighted: 
e The importance of focusing on impact and on evidence and evaluation. These will help to 
deliver better impact. 


Appendix: Notes from table discussions of 
capacity building challenges 


Barriers / blocks to the integration of high quality public engagement in REF submissions 


Issue of resources (time, funding, skills etc.) 


Lack of resources, people, money, time 

Time taken for academics 

Where is the funding for public engagement? Researchers keep the costs down and 
public engagement suffers 

Early career researchers need time and support 

Many institutions don’t have the resources 

Lack of training and tools 

Lack of resource e.g. temp & part-time roles 

Academics feel they haven’t got the funds or budget to do it 

Lack of public engagement resources 

Time: you often hear academics don’t have time particularly if they are working on REF 
research 

Not having the capacity within the University to offer support for instance people on 
short-term contracts 

Need to provide additional institutional support for academics 

Public engagement is not a quick win 


Structural issues 


Institutional structure, if you don’t have leads or offices 

There is a requirement for culture change across institutions to improve their approach 
to public engagement 

Processes institutions don’t really fit 

Lack of institutional recognition 

Lack of recognition of professional service staff (experts) at senior level 

Lack of awareness of available support, lack of recognition of skills value of input 
professional service support for public engagement from senior management and those 
planning the REF process 

Short-term academic contracts which don’t help build relationships 

Public engagement funding rarely pays the salary or time 


Issues of language/definitions 


People are not used to the language of public engagement, both internally and 
externally 

Where are the boundaries between public engagement and knowledge exchange et 
cetera? A lot of public engagement activity is focused on the individual, not the research 
People are not used to the language of public engagement 

Definitions of public engagement 

Ongoing perception that public engagement needs to be highly visible, like Brian Cox: if 
it’s not highly visible then it’s not going in 


Lack of clarity about what is public engagement, what is dissemination, what is engaged 
research: boundaries 

For many institutions engagement is about increasing student numbers 

Lack of institutional understanding as to what impact public engagement is 


Challenge of evidencing impacts from PE 


What evidence can you provide to prove good public engagement impact 

How do you show it makes a difference 

Sometimes people don’t incorporate public engagement in case studies because it is so 
hard to evidence 

Too difficult to evidence or articulate 

Fixation on reach (e.g. Audience numbers, but significance?) 

How to measure the link between research, engagement and impact 

Need for more resource around evaluation: is this the role and responsibility of the 
University? 

Challenge being able to evaluate with rigour for public engagement if you do not have 
the expertise in this 

Ethics of asking people for evidence (e.g. Do you feel better) did you get ethics to 
evaluate appropriately before you started the project for instance providing usable 
evidence of how information is collected 

Evidence: understanding what good evidence looks like, particularly that which shows 
significance 

Lack of evidence with regard to significance and impact 

Need examples of what robust evidence looks like 

Support for good evaluation 


(Perception) PE not valued/seen as risky 


The will to back it 

Public engagement is not seen as important enough to allow researchers to invest the 
time required for high quality public engagement 

A perceived hierarchy of impacts with economic greater than public engagement 
Culture of research values commercial and economic impact 

Many feel public engagement could never lead to real impact 

Fear of losing marks if public engagement is included, especially if the institution has so 
many great impact case studies already 

Perception of what panels will think still an issue at an institutional level 

There is a perception that public engagement is special compared with the more familiar 
policy business engagement which are more tangible 

Faculty led belief that public engagement is not well received by REF panels 

Less confidence by faculty who make decisions on REF case studies 

High-level support is absent for public engagement is not seen as a significant impact 
relative to other types e.g. Engineering and health 

Academics are making a huge emotional investment and risk to reputation 

Subject specific consideration of the worth of public engagement: concern the impact 
case studies based on evidence of public engagement will not be assessed strongly in the 
units of assessment not traditionally based around this type of activity 

Public engagement is not impact - this is a common institutional review 

There is still an enduring narrative that public engagement is weak 


Need for additional guidance/examples 


No sector agreement on what high-quality public engagement looks like 

Lack of guidance 

Lack of clarity over how and what the panels will value in public engagement and 
therefore not including the narrative 

We need examples of impact case studies that have public engagement embedded and 
integrated throughout 

Need for guidance in the environment statement about forward plans for instance 
innovation, diverse city of public engagement, recognition mechanisms for engagement 


Other issues 


A cultural voluntary commitment to public engagement 

People doing the best public engagement and not necessarily the ones with good 
underpinning research 

Public engagement is seen as an add-on people’s day job 

Academics sometimes don’t value non-academic staff 

Neglects areas of research were public engagement is typically one way, by necessity: for 
example, arts and humanities 

Research discipline bias 

Direction is always informed by academics 

The link to research outputs 

Other audiences are a more obvious fit research. Academics with limited time prioritise 
the easiest and most accessible audience 

Dominance of communications and public relations (high reach) 

Lack of good public engagement anyway, with no consequence from funders 
Overreliance on easier methods 

Knowledge of public engagement activities at the institution by those who are compiling 
the case studies 

Competition perceived between institutions which makes collaboration challenging 
Needs to be implemented at the start, earlier in the funding cycles 

Sometimes seems like a bolt on as you parachute in. Almost can seem like a knee-jerk 
reaction 

Reluctance to engage is don’t understand the benefits 

Research outputs progress individuals careers but impact case studies don’t 

How to get visibility of public engagement in amongst all other forms of engagement 


‘Triggers’ or methods you have deployed which have been successful 


Reward, recognition, career progression 


Awards and recognition 

Career progression needs to recognise the value of public engagement 

reward and recognition 

Public engagement is written into workload model 

Inclusion in hiring and promotion process 

Having public engagement written, as an option, into researchers contracts e.g. They are 
allowed to do up to X days of public engagement a year 

Reputation and awards 


Accreditation 


Training/researcher development 


Masterclasses on theory et cetera 

Should we focused training and development of public engagement on early career 
researchers 

Research development support needs to expand support this activity so that academics 
aren’t doing the logistical background work 

Commitment to early career researcher training, catch them at the start 

Training 

An induction pack the new starters 


Champions/leadership 


Get academics involved in public engagement to talk to other academics 
Influence early career researchers 

An impact champions network 

Lots of internal supporters of research are involved 

Teams which do public engagement well coaching other teams or researchers 


Communication/framing 


Getting the narrative out about public engagement and that it matters to them across all 
panels 

Ensuring we have systems to share projects and learn 

Value public engagement in impact as a matter of course do not encourage a narrow 
focus on REF 

Building capacity within universities can translate to senior leaders, impact managers et 
cetera about what public engagement is 

Framing audiences as external, not public 

Having a clear idea as to why people are doing engagement: they need to articulate this 
Nice analysis and report showing public engagement is embedded 

Talking to individual researchers to try to explain how public engagement adds value 


Funding 


Public engagement is written into collaborative doctoral training proposals 

Funding mechanisms 

Internal seed funds: make them define their audience, practice, process et cetera 

Being clear about the economic value of impact case studies 

Recognition of the potential natural gain of a four-star case study in relation to justifying 
resource 


Support 


We have a group of impact ninjas 

Help with link to research and engagement activities processes 

Public engagement officers role is to do project management and coordination 
Matchmaking, disseminating work, introducing new partners 

Get impact and public engagement people working together and aligned 

Add value - enhance what they’re doing 

Getting people to think about impact building to their proposal from the beginning 


e Sharing of wider evaluation team 

e Public engagement team involved in the REF readiness exercise 
e Having externals, Publics read the impact case studies 

e Evaluation 


Case studies and examples 

e We need a variety of case studies to show the range 

e Giving information about the many different types of impact that public engagement can 
have beyond informing, which is as far as some people think it goes 

e Use case studies to highlight the benefit and added value and to give them ideas 


Ideas for how the NCCPE and other agencies might provide well targeted support 


Support to panels 
e Provide support to REF panels in assessing public engagement, or help public 
engagement staff get involved in that conversation 
e Clear and accessible guidance 
e Training for REF panel members to ensure baseline understanding 


Support to sector/institutions 
e Watermark, EDGE tool 
e Simplify the language 
e Feedback on developing case studies 
e Database of public engagement funders, e.g. Small pots et cetera 
e Promote small pots of funding which are often underused 
e Provide a hub for good practice 
e Having information and resources that academics can access through other external 
agencies such as research councils and NCC, all available on the website 
e Sharing between universities via critical friends 
e More fora to discuss 
e Guidance for external partners, e.g. Museums 


Support to PEPs 
e A barrier is the lack of a baseline: help us to find this 
e More studies like public attitudes to science, which we can use as baselines REF impact 
case studies Support to senior leaders 
e Something we can take back and share 


Training/CPD 
e Focus on early career researchers 
e Early career researcher focus 
e Training the gathering of impact evidence 
e more training for early career researchers on impact and engagement 
e Training on establishing and evaluating dissemination impact, e.g. Online and media 
e CPD the public engagement professionals and others (i.e. Impact people and senior 
managers) 


NCCPE should work with groups like the leadership foundation and higher education 
Academy to incorporate public engagement in leadership training the prospective vice 
chancellors and pro-vice chancellors 


Advocacy 


Clarify the value of public engagement at leadership level where decisions are made 
about which case studies are submitted 

Lobby to ensure that people who review evidence to understand what robust evidence 
looks like and/or encourage those specialists have a weighted voice in the assessment 
process 


Setting the standards/providing examples 


Explain what ‘high quality’ means to academics 

Examples with good quality evidence and bad, the different types of evidence 

More research into public engagement, distinct from evaluation of activities or programs 
and/or tying this evaluation into research about public engagement: we need to show 
evidence that the activities are doing can bring impact (e.g. Cite a peer reviewed paper) 
and compare your work to this, not necessarily do a full research paper on your own 
project 

Examples of best practice in evidence and engagement 

Examples of robust evidence, perhaps from the last REF but not actually in an impact 
case study 

A mature case study will speak to process, structure and outcomes 

Evidence of what good public engagement looks like 

Could we do a pilot exercise which would signal changes 
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